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The following four channels are considered: (A) a class of discrete 
memoryless channels with q inputs and outputs, (B) the time-discrete, 
amplitude-continuous memoryless channel with additive Gaussian noise 
and amplitude constraint, (C) the same as channel B but with energy 
instead of amplitude constraint, (D) a class of time -discrete, amplitude- 
continuous memoryless channels with amplitude constraint and non- 
Gaussian noise. For each channel the theoretical capabilities of "bounded 
discrepancy decoding" are studied. 

The "discrepancy" between two vectors is a distance or distance-like 
quantity defined such that the optimal decoder is a "minimum discrepancy 
decoder." For example, for channel A the discrepancy is the Hamming 
distance, and for channel B the discrepancy is the Euclidean distance. 
Bounded discrepancy decoding is a nonoptimal decoding scheme in which 
disjoint regions in the space of possible received vectors are constructed 
about each code word, each region consisting of those vectors within a 
fixed discrepancy of that code word. For example, in channels A and B 
the regions are spheres with centers at the code words and radius d/2 where 
d is the minimum distance between code words. If the received vector is in 
the region about code word i, it is decoded as code word i; otherwise the 
decoder announces an erroi'. 

For all four classes of channels the following is shown to hold: There 
exists a fixed positive rate C B below which it is possible (asymptotically in 
n) to obtain exponentially small error probability using bounded discrep- 
ancy decoding. In many cases C B is shown to be strictly less than the channel 
capacity. 
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i. introduction 

To fix ideas, let us consider first the special case of coding for the 
binary symmetric channel. A code is defined as a set of M binary n-vectors 
x = (xi , x 2 , . . . , x n ) where x k = or 1 (k = 1, 2, . . . , n). The indi- 
vidual vectors are called code words. The transmission rate R is defined 
by M = 2 nR . The Hamming distance between two binary n- vectors is 
the number of positions in which they differ. 

The code words are transmitted through a noisy channel. The re- 
ceived vector y is a binary w-vector whose fcth coordinate is 

Vk = x k + z k (mod 2), k = 1, 2, . . . , n, (1) 

where x k is the /cth coordinate of the transmitted code vector, and the 
Zk (k = 1, 2, . . . , n) are statistically independent random variables 
which assume the value 1 with probability p o (0 ^ p ^ |), and the 
value with probability 1 — p . Thus p is the probability that a given 
bit is received in error. This channel is the "binary symmetric channel." 
It is assumed that each of the M code words is equally likely to be 
transmitted, and it is the task of the decoder to examine the received 
vector y and decide which code word was actually transmitted. We are 
interested in two types of decoding. 
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The first is termed minimum distance decoding or minimum discrepancy 
decoding (MDD), and here the decoder selects that code word which 
has the smallest Hamming distance from the received vector, and an- 
nounces that word as the one which was transmitted. It is not hard to 
show that MDD is optimum in the sense that it minimizes the average 
probability of error for a given code. Let us denote by P eM the average 
probability of error using MDD. The Fundamental Theorem of Infor- 
mation Theory 1 states that for any rate R less than the channel capacity 
C = 1 + p„ logo p + (1 — Vo) log-- (1 — Po), there exists a sequence of 
n-dimensional codes (one for each n) with rate R such that P eM — * 0, 
as n -> oo. Further we may write P eM = 2- nB(ft)+o( " ) where E{R) > 
when R < C. Estimates of the exponent E(R) have been found. 2 ' 3. 

In order to construct specific codes many workers (for example, see 
Refs. 5 and 6) have considered codes in which the minimum Hamming 
distance between code words is d. Such codes are capable of correcting 
errors affecting e = (d — l)/2 or fewer coordinates. Suppose that the 
code under consideration has minimum distance d and that the decoder 
corrects only errors corrupting e = (d — 1 )/2 or fewer coordinates (and 
announces an error if the received vector is not within Hamming dis- 
tance e of some code word). We term this type of decoding bounded dis- 
tance decoding or bounded discrepancy decoding (BDD) and the resulting 
error probability P eB .f Since BDD does not exploit the full error- 
correcting potential of the code (an error may corrupt more than e = 
( r / _ i)/2 coordinates and still be correctable using MDD) it is clear 
that P cB ^ PeM . In this paper we shall study the theoretical capabilities 
of BDD, and show quantitatively what is lost by using BDD instead 
of MDD. 

For the binary symmetric channel the following will be shown to 
hold: 

Theorem A: There exists a fixed rate C B (called the bounded distance 
decoding channel capacity) below which it is possible (asymptotically in 
n) to obtain exponentially small error probability using BDD. In other 
words, for every R < C„ , there exists a sequence of n-dimensional codes 
(one for each n) with rate R such that P eB = 2~ nEB( ' {)+o( " ) where E B (R) > 
OifR <C B ). Further if R > C B , P eB -> 1 as »-»«, 

Although C B is not known exactly, it can be shown to satisfy 



t The Peterson-Chicn algorithm fur decoding Bose-Chaudhuri-Hoequenghem 
codes is an example of BDD. (See Chien, R. T., Cycling Decoding Procedures 
for Bose-Chaudhuri-Hoequonghrrn Codes, IEEK Trans, on Information The- 
ory, IT -10, 1964, pp. 357-363). 
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where H(p) = — p log., p — (1 — p) log 2 (1 — p), and p is the bit error 
probability of the binary symmetric channel. These upper and lower 
bounds on C B are plotted vs p in Fig. 1. It is clear that C B is bounded 
below the channel capacity C, the maximum "error-free rate" obtain- 
able using (optimum) MDD. The exponent E B (R) can also be esti- 
mated by upper and lower bounds. 

In this paper we shall study a number of different channels (continu- 
ous as well as discrete). For each channel we shall define a distance- 
like function called the "discrepancy" which will be chosen so that 
the optimum decoder is a "minimum discrepancy decoder." (For the 
binary symmetric channel the discrepancy is the Hamming distance, 
and in most of the cases to be considered the discrepancy is a metric.) 
We then define a "bounded discrepancy decoder" and compare the 
capabilities of BDD to those of optimal MDD. In all cases we will 
deduce the existence of a "bounded distance decoding channel capacity" 
C B for which Theorem A holds. In many of these cases we will show 
that C B is strictly less than the channel capacity. 

A glossary of symbols is included at the end of the paper. 
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Fig. 1 — Upper and lower bounds on C B (2) (in bits) for binary symmetric 
channel (solid lines). Thus C« lies in the shaded area. The channel capacity C 
is the dotted line. The equivalent value of Cb corresponding to natural logarithms 
is given in parenthesis. 
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II. SUMMARY OF RESULTS 

We shall consider four classes of channels. In each case the input and 
output are n-vectors x = (ah , .r 2 , • • •, x n ) and y = (iji , y 2 , ■ ■ •, y n ) 
respectively, related by 

y k = x k + z k , fc = 1, 2, •■• n. (3) 

The symbols x k , y k and the noise digits z k are as follows: 

Channel A (Discrete Channel): The input digits x k (k = 1,2, • ■ ■ ,n), 
the output digits y k (k = 1, 2, ■ ■ ■ , n), &nd the noise digits z k (k = 
1, 2, • • • , n) are members of the finite alphabet of q symbols, 0, 1, • • • , 
3—1. The addition in (3) is modulo q. The z k are independent random 
variables assuming the value with probability 1 - p , and the values 
1, 2, • • • , q - 1 with probability p a / (q - 1). Thus the channel trans- 
mits each symbol correctly with probability 1 — p a , and makes an 
error with probability p , all errors being equally likely. The Hamming 
distance d w (u,v) between two n-vectors u and v with entries from the 
alphabet of q symbols is the number of positions in which u and v differ. 
Channel B (Gaussian Chamiel with Amplitude Constraint): The 
digits x k ,y k ,z k (k = 1,2, ■ ■■ ,n) are real numbers. The input vector 
x satisfies an amplitude constraint : 

-A ^ x k ^ +A, k = 1,2, .-.,71. (4) 

The noise digits z k (k = 1, 2, •• • , n) are independent Gaussian random 
variables with mean zero and variance N. The Euclidean distance be- 
tween two vectors u and v is denoted by d E (u,v). 

Channel C (Gaussian Channel with Energy Constraint): The digits 
Xk Vk , Zk (k = 1, 2, ■ • ■ , n) are real numbers. The input vector x lies on 
the surface of the n-dimensional hypersphere with center at the origin 
and radius y/nP. Thus 

k=\ 

As in channel B, the noise digits z k (k = 1,2, • • • , n) are independent 
Gaussian random variables with mean zero and variance AT. The signal 
"energy" is ^x k = nP, and the expected noise "energy" is 

k 

so that the signal-to-noise energy ratio is P/N. This quantity is also 
the ratio of signal-to-noise "average power." 
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Channel C is of course closely related to the bandlimited channel 
with white Gaussian noise. 1 Such an identification, however, must be 
made with care, and we shall sidestep the issue here. 

Channel D (Continuous Channel with Amplitude Constraint): The 
vectors x, y and z are members of e„ defined as the set of n-vectors 
u = (wi , Ut , • • •, u n ) where the coordinates Uk (k = I, 2, • • • , n) are 
real numbers satisfying 

-A ^ u k ^ A. (6) 

We shall assume that the symbols "-J-" and " — " when applied to 
coordinates of vectors in e„ denote addition and substraction modulo 
2A, with the result reduced into the interval [—A, A], Equation (3) will 
thus be rewritten as 

Vh = %k + z k , k = 1, 2, • • • , n. (7) 

The noise coordinates Zk are assumed to be independent identically 
distributed random variables with probability density function p{u) 
which satisfies: 

(a) p(w) = 0, | u\ > A. 

(b) p(u) > 0, \u\ ^ A. 

(c) p{u) is an even function of u. (8) 

(d) p(u) is a continuous, strictly monotone decreasing function 
of u for ^ u ^ A . 

(e) There exists an a > such that for small u we may write 

p{u) = p(0)[l + O(u a )}. 

Thus what we have done is to wrap the interval [ — A,+.A] onto the 
circumference of a circle, and assume that the noise perturbs each 
coordinate along the circumference a distance z* ( — A ^ Zk ^ A). Such 
a channel is reasonable for the case where the x k correspond to the 
phase of a fixed waveform, f and also as an approximation to other 
channels. 

For each channel we define a code as a set of M n-vectors x satisfying 
the above constraints. The transmission rate R is defined by R = 
(1/n) In M\ so that M = e nR We assume that each of the M code words 
is equally likely to be transmitted. It is the task of the decoder to exa- 
mine the received vector y and to decide which code word was actually 

t An example in which this model iB applicable may be found in A. J. Viterbi, 
"On a Class of Polyphase Codes for the Coherent Gaussian Channel," IEEE 
International Convention Record, part 7, 1965, pp. 209-213. 

t For the remainder of this paper all logarithms will be taken to the base e. 
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transmitted. If P e , is the probability that the decoder makes an incor- 
rect choice when code word i is transmitted (i = 1, 2, 3, • • ■ , M), and 
if each of the M code words is equally likely to be transmitted, then 
the overall probability of a decoding error is 

Pe = (1/lOE-P*- (9) 

i=l 

The optimal decoder is defined as the decoding system which minimizes 
P e for a given code. 

As was done for the binary symmetric channel in Section I we shall 
consider two types of decoding. 

Channel A: The optimal decoder may be shown to be the one which 
selects that code word x which minimizes the Hamming distance, 
rf„(x,y) between x and the received vector y. Accordingly, we define 
the "discrepancy" as the Hamming distance, and the optimal decoder 
is the minimum discrepancy decoder (or minimum distance decoder) de- 
noted by MDD. Let us denote by P eM the probability of error (P e ) 
using MDD. 

The channel capacity of Channel A is readily shown to be 

C = C(p ) = In q - H(p„) - Vo In (q - 1), (10) 

where 

H(p) = - P In p - (1 - p) In (1 - P ). (11) 

The Fundamental Theorem of Information Theory 1,7 states that for 
any R < C there exists a sequence of n-dimensional codes (one for each 
n) such that P eM = e - nE{R)+oln) (where E(R) > when R < C). Further 
if R > C, PeM ^ 1 so that C is the supremum of those rates for which 
it is possible to obtain vanishingly small error probability using MDD. 
The second type of decoding is described as follows: For the code 
being used, let d be the minimum Hamming distance between pairs of 
code words. About each of the M code words we construct a "sphere" 
in the space of q n n-vectors, consisting of those vectors not more than 
Hamming distance (d - l)/2 from that code word. All these spheres 
are disjoint. If the received vector is in the sphere about code word i, it 
is decoded as code word i. If the received vector is in no sphere, then the 
decoder announces an error. We term this type of decoding bounded 
discrepancy decoding (BDD), and denote the resulting error probability 
by P eB . (i.e., P,h is the probability that the received vector is not in 
the sphere about the transmitted code word.) Alternately, the bounded 
discrepancy decoder corrects errors affecting up to e = (d — l)/2 
positions and no more. Clearly P eB ^ Peu . 
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In connection with BDD we are interested in the quantity M(n,d), 
the maximum number of code words in an ri-dimensional code with 
minimum distance d. The corresponding transmission rate is R(n,d) = 
(1/n) In M(n,d). The following bounds hold: 

Ford/2n > (q - l)/2q: 



M(n,d) ^ 



P 



m 



(12a) 



Ford/2n < (q - l)/2q: 



q" 



d-1 

Sir 

where 



(q-lV 



^ M{n,d) ^ 



n[l-(2g/g-l)/3] 



qd 
nq"K((S) 



id/2] 

S 

r=0 



©<--«'0f-')" 



p = d/2n, 



K05) =Mi -tfiq/(q- DL 



(12b) 

(12c) 
(12d) 
(126) 



and where [x] denotes the largest integer not greater than x. The upper 
bound (12a) and the first upper bound of (12b) are the well known 
Plotkin bounds, 8,9 and the lower bound of (12b) is the well known 
Varshamov-Gilbert-Sacks bound as given in Ref. 8; the second upper 
bound of (12b) is established in Section III. 

Now let us let n and d become large while the ratio /3 = d/2n is held 
fixed, and define R(P) = lim R(n,d) = lim R(n,2(in). We obtain from 



(12a): 

B(fi) = 0, P> (q- l)/2g, 
and from (12b): 
In q - H(2p) - 2/3 In {q - 1) g R{0) 



< 



f( l "A) 



In q, 



(13a) 



(13b) 



Ing - H(t0) - tfihi (q - 1), 

where H (p) is defined by (11). The second upper bound in (13b) is the 
same as the Elias bound which was obtained independently. Although 
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the bounds of (12) and (13) are of interest in themselves, we make 
use of them here to demonstrate the following: 

Theorem A: There exists a fixed rate C B , called the "bounded discrepancy 
decoding channel capacity," such that for any rate R < C B , there exists a 
sequence of n-dimensional codes (one for each n) such that P eB = exp 
\-nE B (R) + o(n)] (where E B (R) > for R < C B ). Further if R > 
C B , P cB JL 1, so that (',< is the supremum of those rates for which it is 
possible to obtain vanishingly small error probability using BDD. 

For channel A we shall show 

C(2p n ) ^ C B ^ C(tpo) < C(Po) = (', (14) 

so that C B is strictly less than C the "maximum error free" rate using 
MDD. 

Finally we can estimate E B (R) by 



« (| . V°) 



^ E B (R) ^ 



a G[ 1 -2(^T) s ]' Po )' 

(q ~ l[H( s) +sln (g~ D l A 



(15) 



where s = s(R) is defined by 

R = C(s) = In q - H(s) - s In (q - 1), (15a) 

and 

a(p, Vo ) = p In ■£ + (1 - p) In ^ ~ p) . (15b) 

Channel B: For this channel it may be shown that the optimum de- 
coder minimizes the Euclidean distance rf B (x,y) between the received 
vector y and the code word x. Accordingly, we define the discrepancy 
as the Euclidean distance d E , so that the optimal decoder is the minimum 
discrepancy decoder (MDD). Here too the channel capacity C is the 
maximum rate below which it is possible to obtain vanishingly small 
P eM . An exact expression for C is not known but it has been estimated 
by upper and lower bounds by Shannon 1 and a method for computing 
C is outlined by Wolfowitz. 10 Bounded discrepancy decoding (BDD) is 
defined exactly as for Channel A with the Euclidean distance used 
instead of the Hamming distance. 

Let M(n,d 2 ) be the maximum number of points in an n-dimensional 
code with minimum distance d, and let R(n,d 2 ) = (1/n) In M (n,d 2 ) be 
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the corresponding transmission rate. We let n and d become large while 
the ratio /S = (d/2) 2 /n = d 2 /(4n) is held fixed, and define B(fi) = 
lim R(n,d 2 ) = lim R(n,4@n). Let J3 = fi/A . The following estimate of 

n->oo n-»oo 

R(P) is obtained: 

R L (J3) ^ R(0) ^ Ru((3) (16) 

where 



Rvifi) = ■ 






e^h 




(21n2)(l - 2/3), 


\£&<h 




k 2 

In lc (1 °a) 


l* ' < (k 
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W _ 2 U Ph 


- I) 2 




(k = 3,4,5, 


•••) 



(17) 



and 

_ ,_. /In 2 + In 08) + (1 - /§) In (1 - $) - B u 

fi.OJ) = maxj Co(4/3) _ ^ (18) 

where C (£) is defined by 

C.ft) = ln2A/C.({) - *X(*), (19) 

and where X (£) is defined by 



where 



and 



( r(u)e- MiMu) du = £ / e- MVr(u) du, (19a) 

•>0 "0 



r(«) = u, (19b) 



X(f) = \f <T X( < Mu) dJ l . (19c) 

It is verified in Appendix A that for < $/A ^ f , there exists a 
unique X(£) satisfying (19a). The first value of the lower bound R Li is 
dominant for 0.02 ^ /3 < 0.5, and the second R L 2 for < $ ^ 0.02. 

We make use of the estimate of R (/3) (16) to establish Theorem A for 
Channel B. Here we have 

Rl(N) ^ C B ^ R„(N). (20) 
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For large values of A 2 /N, 

C B ^ C -In 2 + *(A 2 /N), (21) 

where C is the channel capacity, the maximum "error free" rate using 
MDD, and e — » as A 2 /N — * °o . Thus for large values of the "signal-to- 
noise ratio" A 2 /N, C B is within a constant of C, so that the ratio C B /C — » 
1 as A 2 /N — * oo . An estimate of E B (R) is also obtained. 

Channel C: As with Channel B, the optimal decoder is the decoder 
which selects that code word x which has the smallest Euclidean distance 
from the received vector y. Thus if y = {y x , y% , ■ • ■ , y n ), the decoder 
announces that code word x which minimizes (with respect to x) 

d B (x,y) = t, Oft - VxY = E ft* + £ V* ~ 2 L *to • ( 22 ) 

t=i t *; fc 

Since 2 ^ = nP > d E (x,y) is minimized when ^x k y k is maximized. 

-t * 

Hence optimal decoding is equivalent to selection of that code word x 
which minimizes the angle a(x,y) between x and y, where 

I] XkVk 
cosa(x,y) = == — ^x^ — ni ■ ( 23 ) 

Thus if we define the discrepancy between x and y as the angle a(x,y), 
the optimal decoder is the minimum discrepancy decoder (MDD). Let 
us denote by P eM the error probability using MDD. 

The channel capacity is C = § In [1 + (P/N)], and is the maximum 
rate below which it is possible to obtain vanishingly small P eM . Further 
for any R < C, there exists a sequence of n-dimensional codes such that 
p m = e -«*(K)+°<«) Estimates of E{R) are obtained in Refs. 11 and 12. 

The bounded discrepancy decoder (and P eB ) is defined exactly as for 
Channels A and B but with the angle a(x,y) used instead of the Ham- 
ming or the Euclidean distance. 

In connection with BDD we consider M(n,6), the maximum number 
of points in an n-dimensional code with minimum angle 0, and the 
corresponding rate R(n,d) = (1/n) In M (n,0). The following bounds 
hold for < tt/2: 



(4-3 



- 1 



^m^ix^'^t 
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^ M(nft) (24) 

■\ZttY f — - — J sin \p tan \f/ 

2r (|J y (sin <p) n ~ 2 (cos p - cos fldp 

where \f/ = sin -1 \/2 sin (0/2). The upper bound was obtained by 
Rankin, 13 and the lower bound is obtained in Section V. If we let n 
become large while 6 is held fixed and let R(6) = lim R(?i,6) we can 

n-»oo 

obtain from (24) 

-In sin 6 ^ R(6) ^ - In y/2 sin (9/2). (25) 

Inequalities (25) will be used to establish Theorem A for Channel C. 
Here we have 

C - In 2 - \ In (1 - e~ 2c ) ^ C B ^ C - £ In 2. (26) 

Estimates will also be obtained for the exponent E B (R) and compari- 
sons to the estimates of E(R) will be made. 

Channel D: For this channel we shall find the optimal decoding 
scheme by proceding as follows. Define the function r{u) by 

r(«) = iln ^M , -A £ u s£ +A (27) 

X p(u) 

where p(u) is the noise probability density function which satisfies 
assumptions (8), and X is a constant to be specified later. Equation (27) 
is meaningful since by (8b), p(u) ^ 0. From (27) we see that 

p(u) = p(0) exp [-Xr(w)] = K exp [-Xr(w)], (28) 

where K = p(0). The n-fold joint probability density for the n inde- 
pendent noise coordinates is 

n n 

Pn(«i , w 2 , ••-, u„) = JJp(u k ) = K n exp [— XJZ »*(«*)]« (29) 
t=i fc=i 

Let us now consider the decoder. Suppose that the received vector is 
y. It is not hard to show that the probability of incorrect decoding is 
minimized when the decoder selects that code word x which maximizes 
p(y|x), the conditional probability density of receiving y given that x 
is transmitted. This quantity is 
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p(y | x) = p n (yi — x\ , Vi — x 2 , • • -,y n — Xn) 

(30) 
= K n exp[-\J2r(yk^x k )]. 

The subtraction of coordinates y k - x k in (30) is performed modulo 2 A 
with the result reduced into the interval [—A,+A\. Thus for a given y, 
the optimal decoder selects that code word x which minimizes 

d„(x,y) = 2>(?/ fc - a*)- (3D 

The function d (x,y) denned on e„ X C„ will be denned as the dis- 
crepancy function, so that the optimal decoder is the minimum dis- 
crepancy decoder (MDD). Denote the resulting error probability by 

P.m. 

Let us remark at this point that the discrepancy has the following 

properties : 

(a) d u (x,y) ^ 0, with equality if and only if x = y. 

(b) d (x,y) = d (y,x). 

It is not necessarily a metric, however, since the triangle inequality 
need not hold. 

For any vector a £ 6„ , let the "region" £„(a,p) be the set of vectors 
($ 6 Qn satisfying d„(a,&) < p. We say that a code has discrepancy p 
if the regions S„(x,p) about all M code words x are disjoint. 

We now describe another, though nonoptimum decoding technique. 
Let p be the largest number such that the code under consideration has 
discrepancy p . Hence the regions £„(x,p ) for all code words x are 
disjoint. If the received vector y € S„(x,p ) for some code word x, then 
it is decoded as x. If y belongs to no region, an error is announced. We 
term this type of decoding bounded discrepancy decoding (BDD), and 
denote the resulting error probability by P eB . Clearly P eB ^ PeM . 

A case of special interest is that for which p (u) = K exp (— Xw 2 ).This 
channel is similar to channel B when X is large (so that the effects of 
wrapping the interval [—A,+ A] onto a circle are minimized]. In this 
case r(u) = u . 

Suppose we are given a function r(u) defined on [— A,+A]. This 
function defines a discrepancy which is appropriate for the class of 
noise densities p(u) = K exp [— \r(u)]. Now a given member of the 
class could be specified by the parameter X. (K is then determined by 
setting the total mass of p(u) equal to unity.) It is convenient instead 
to specify a given member of the class by the parameter N denned by 
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N = E[r(z)] = j r(u)p(u)du = f r (u) # e~ Xr(u W (32) 

That is, given the parameter N corresponding to a X ^ 0, and the func- 
tion r(u), one can solve (32) for X and K . Thus r(u) and N specify 
the channel. For example if r(u) = u, then N = E[z] is the average 
noise "power." It is shown in Appendix H that the channel capacity, 
the maximum "error free" rate using MDD is 

C = C (N) (33) 

where the function C (£) is defined by (19) with the appropriate func- 
tion r(u). It is shown in Appendix A, that C (£) is well defined for 



< £ g 4 / r(u)du. 



A 

Let us now consider BDD. Two important quantities here are M (n,p) 
the largest number of code points in an w-dimensional code with dis- 
crepancy p, and the corresponding rate R(n,p) = (1/n) In M(n,p). We 
let n and p become large while the ratio /3 = p/n is held fixed, and then 
define R (0) = lim R (n,p) = lim R (w,j8n). It is shown in Section VI that 

n-»°o n-»w 

Co&iP) ^ R(fi) ^ C o 03), (34) 

where Co(£) is defined by (19), and 77 is defined by 

r(Wi -f U2) /oe\ 

V = sup ; . (35) 

The addition in (35), Wi + u 2 , is modulo 2A, with the result reduced 
into the interval [—A, -{-A], It is shown in Appendix B that 77 is finite. 
The estimate (34) of R (/?) is used to establish Theorem A for channel 
D. Here C B can be estimated by 

C (2 V N) ^ C B ^ C (N) = C. (36) 

For the special case of the quadratic discrepancy r (u) = u , the quan- 
tity 7} = 2, so that the left-hand member of (36) is C(4N). It will be 
shown that in this case C B is bounded above by C (2N) so that 

C (4N) ^C B ^ C (2N) < C (N) = C. (37) 

Hence in this case C B is strictly less than C. Further, both the upper and 
lower bounds of (37) will be refined for small values of the "signal-to- 
noise" ratio A 2 /N. 

For large values of "signal-to-noise ratio" A 2 /N, (37) becomes 
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I ,„ l £ + , (f) ^„g-J i„ 1 £ + <2 (£) , (38) 

2 2ireN \N / 2 re N \N / 

where ti , t 2 — > as .4 2 /N — > °o . It will follow that C B is within a con- 
stant of C, so that the ratio C B /C — > 1 as A 2 /W — > » . 

III. CHANNEL A (DISCRETE CHANNEL): 

3.1 Lower Bound on M(n,d) 

Our first task is to obtain the second upper bound on M(n,d) of 
inequality (12b). We need the following lemmas: 

Lemma 3.1: Let g\ ,g-> , • • ■ , g„ be real numbers. Then 

£ g>* * 1 (E g k ) 2 - (39) 

a— i n 

Proof: From the Schwarz inequality 

V=i / V=i / V=i / fc-1 

Lemma 3.2: Given a code with minimum distance d, let x, = (xa , x& , 
•••,:»',»), i = 1,2, • • • , m be any set of m points from the code. Let z be 
any n-vector and n (i ■» 1, 2, • ■ ■, m) the Hamming distance d tf (x t , z) 
from x, to z. Then 

(fc) _ a<« - D" (£z\ + kni> »(» - 1) * s o. (40) 

\ n / q \ n / q n 

Proof: Without loss of generality assume z = 0. Arrange the m code 
words in an array 

Xl = .1*11 > 3*12 , • • • , X\ n 
X m = .fml » 3"m2 j * ' ' j X mn . 

Denote by s jk (j = 0,1, •••,<?— 1; fc = 1, 2, • ■ •, n) the number of 
times symbol j appears in column k. Then, since the code has minimum 
distance d, 

(m\ " 9 ~ 1 

n ) d g £ d«(x r , x,) = 2E $•*(« - Sjt) 
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k j k i 

Now 2 X s >* = mn so tnat 

k j 

Since s * = w — £ s jk , by Lemma 3.1, 

£ sot 2 ^ - E (w - E »*) = - [mn - E E s;*] 2 - 
Also by Lemma 3.1, 

n g-1 j 

E Z s Jfc 2 ^ t- — 7y- E Z s Jfc ] 2 . 

a-=i ;=i (g — \)n k j>o 



(42) 



(43) 



Observing that Z 2 s tf = E r * . and substituting (42) and (43) into 
(41), we obtain 



/ra\ , ^ mn 1 / V* .. V 



(44) 

On dividing (44) by nq/2(q - 1), the lemma follows. 

Derivation of the Bound: 

Let us assume that we have an w-dimensional code with minimum 
distance d(d/2n < (q - l)/2q) with M = M (n,d) code points. Con- 
sider the "sphere" of radius t(d/2) in the space of n-vectors about each 
code point where 



i ( 1 V'-T^T^) 



and = d/2n. (Since t ^ 1, these spheres are not necessarily disjoint.) 
To each point of the sphere at Hamming distance r from the center 
assign weight u>(r) = td/2 — r. The "mass" /z of each sphere is there- 
fore 



Ud/2] 

M = 



gm-xrg-,). 
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If an n-vector z is simultaneously in the sphere about the m code 
words Xi , x 2 , • • • , x,„ , then we assign a weight co 2 to z given by the sum 
of its weights in each sphere, i.e., 

1 = 1 Z i = l 

where r, = rf w (x, , z). If z lies in no sphere co z = 0. Consequently, we 
have 

mass of all ?i-vectors = ^ w z = M(n,d)'P- (48) 

all n -vectors 
z 

We will bound il/ by finding a bound on ^3 "* . Letting s = s z = co z /n, 
(47) becomes 

T r 
i ' mid ia (aq) 

n 2n 

Substituting (49) into (40) we get 

mW - 2mm + s 2 - 2 (g ~ 1} mhfi 

(.50) 

+ 2 i^il ms + 2 ^il ^ - 2( ? - X) 0m ^ 0. 
9 9 9 

Rewriting (50) 



< s < m 



\ 9 9 

(a ^> - a*)] • 



(51) 



_^2 

Since by choice of t (45), 

fp - 2 ( g~ 1} , + 2 ^-Hil = 0, 
9 9 

and 

2 (9 ~ 1} - 20t > when < ^i 
9 2 9 
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(51) can be satisfied only when 



s g i - W (q - i) - * w - 


(52) 


Thus 




£ «,=£«-n ^ K(0)nq H . 

nil n -vectors 

Z 


(53) 


Hence from (53), (48) and (46) we have 




Uf 1\ - K(0)nq» 




M(n,d) ^ ltf in] / v 

g ^(«J»-r)( fl -D- 


(54) 


where 




£ - 3 " * A I A 2 ^ ^ K(B) ~ 




1 V \ X q-l)' KW) l-m/(q- 


-1)' 


and/3 < (q - l)/2q. 





3.2 Asymptotic Estimates of M (n,d) (13) 

Equation (13a) and the first upper bound of (13b) follow directly 
from (12a) and the first upper bound of (12b) by writing R(n,2(5n) = 
(l/n) In M(n,20n) and letting n tend to infinity. The lower bound on 
R(P) of (13b) follows from the lower bound on M (n,d) of (12b) and 
the fact that (Ref. 8, Appendix A) 

lim - In E ( n ) (q - l) r = fffe) - £ In (q - 1). (55) 



►oo n r =o \r 



The second upper bound of (13b) follows from (54) and (55) and the 
fact that 



I ©(H <«-»•='"£"© '.-»•• 



(56) 



In the important special case of binary codes (q = 2), the second 
upper bound of (13b) is always sharper than the first upper bound. Thus 
for q = 2 we have for /3 < |: 

1 - #(2/3) ^ R(fi) £ 1 - H(i - h VT^^). (57) 

These upper and lower bounds converge at /3 = \ yielding R (/3) = 0, 
|3 ^ |. Inequality (57), is plotted in Fig. 2. 
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Fig. 2 (Channel A) — Upper and lower bounds on R(0) for the binary sym- 
metric channel (57). The dotted lines are the best bounds given in Ref. 8. 

3.3 Bounded Discrepancy Decoding Channel Capacity 

When Shannon's Fundamental Coding Theorem is applied to channel 
A, one finds that for every R less than the channel capacity C = 1 — 
H (p ) — p In (q — 1), there exists a sequence of codes (one for each ?i) 
with transmission rate R such that the error probability using MDD, 

PeM —* 0. Further R > C, P eM — > 1. Thus the channel capacity C is the 
supremum of those rates R for which it is possible (asymptotically in n) 
to obtain vanishingly small error probability using (optimal) MDD. We 
now ask what is the largest rate for which it is possible to obtain asymp- 
totically vanishingly small error probability using BDD? 

Let us suppose that for every n, an n-dimensional code is available 
with d/2n = 0. Using BDD we have error probability 



P eB = Pr [number of errors ^ d/2 = fin]. 



(58) 



Since the errors in each digit occur independently with probability p , 
we have by the weak law of large numbers that lim P eB = or 1 according 

as fi > p or fi < p . 

If we define the bounded discrepancy decoding channel capacity, de- 
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noted by C B , as the supremum of the rates for which it is possible 
(asymptotically in n) to obtain vanishingly small P eB , we have from 
the foregoing that C B = R(p a ). Making use of the second upper bound 
on R (/3) of (13b) and the fact that t> 1 for > 0, we have for p > 

C B = R{Vo) ^ 1 - H(tp ) - tp \n (q-l) = C{tp„) < C. (59) 

Thus C B is bounded away from C. 

In the binary case (q = 2), we make use of (13b) or (57) to obtain 
1 - H(2 Vo ) rZC B <,l - H(\ -\\/l ~ 4p„). (60) 

Inequality (60) is plotted in Fig. 1. 

3.4 Exponential Behavior of P eB 

For a fixed R < C B , denote by P eB * the smallest attainable value of 
P eB . It was shown above that P eB * -^ 0. We shall now show that P eB * = 
e - nBfl(R)+0(n) , where E B > and obtain estimates of E B (R). 

Given an n and R, denote by /3 n (#) the largest value of /3 attainable 
for an w-dimensional code with transmission rate R. With R held fixed, 
let p(R) = lim j8 n (/2). Then 0(R) satisfies 

n-*°o 

R L (P(R)) ^ R ^ Ru(P(R)), (61) 

where Rl(J3) and Ru(P) are the upper and lower bounds of (13b). If we 
define the parameter s = s (R ) by 

R = In q - H(s) - s In (q - 1), (62) 

we obtain from (61) and (13b) 

(q - l[H(s) + sin (q - 1) 



| ^ 0(R) ^ i 



l[ H(s) +sln (g - 1) 1 
L In q J 



2Q ' ^ (63) 



f 1 " -£-t\ 

V 2(8 - 1)/ 



Thus for any ^ there exists a code (for n sufficiently large) with 
minimum distance d = fi(R)-2n. With R fixed, this code minimizes 
P eB . Thus from (58) 

P eB * = Pr [no. of errors ^ n(l(R)] 

= t (f)po r (l-Po) n - r . (64) 
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Making use of the fact that (Ref. 8, Appendix A) 

lim - I In £ (") pj (1 - V o) n ~ T = «(p,Po), (65) 



fl-»CO IV T=pn \' / 

where a( P ,p ) = p In (p/p ) + (1 - p) In [(1 - p)/(l - p.)], we have 
from (64): 

E B (R) = - lim (1/n) P eB * = a(0(R),p„). (66) 

n-»oo 

Inequality (63) provides bounds on /3(/2) and hence an estimate of 
E„(R). Let us observe that for R = (s = (g — l)/g) the upper and 
lower bounds on 0(ft) (63) converge yielding 0(R) = (q - l)/2g so 
that 

E B (o) =«(St' Po )' (67) 

Further since a{jp , p ) = 0, E B (R) vanishes when R = R(p ) = C B . 

In the binary case, the second upper bound on 0(R) (63) is always 
sharper than the first, so that (66) yields 



«(£,*) 



^ EAR) ^ a(s(\ - s),p<,). (68) 



Inequality (68) is plotted in Figs. 3(a) and 3(b) for p = 5 X 10 ' 
and p = 10~ 4 respectively. It can be seen from Fig. 3(b) that for certain 
values of R the upper bound on E B (R ) is greater than the lower bound 
on E(R) (the best exponent for MDD). Thus although E ^ E B 
(since P, M ^ Pea), there is nothing to indicate that the strict inequality 
always holds. 

IV. CHANNEL 13 (GAUSSIAN CHANNEL WITH AMPLITUDE CONSTRAINT) 

Our first task is to establish the bounds on ft(/3) given in Section II. 
4.1 Lower Bound on R (/3 ) 

4.1.1 Bound for Large /3 

It would not violate the code constraints if the coordinates of the 
code words were further restricted to be ±A. In this case the code is a 
binary code and the Hamming distance d w (x,y) between two vectors 
x and y is related to the Euclidean distance d E (x,y) by 
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Fig. 3(a) (Channel A) — Upper and lower bounds on the exponent Eb{R) for 
the caae q = 2 with p„ = 0.05 (solid lines). Upper and lower bounds on E(R) are 
in dotted lines. 

Fig. 3(b) (Channel A) — Upper and lower bounds on the exponent E B (R) 
for the case q = 2 with p„ = 10 4 (solid lines). Upper and lower bounds on E(R) 
are in dotted lines. 



d„(x,y) = rf E 2 (x,y)/4A 2 . 



(69) 



Thus if a code (with coordinates restricted to ±A) has minimum 
Hamming distance d a = d/4A 2 , the minimum Euclidean distance is d. 

Thus /3 = 0/A 2 = d/4A 2 w = d H /n. 

Now let R H (n,dn) be the maximum rate for which a binary n-dimen- 
sional code with minimum Hamming distance d H exists. We let n, and 
d H become large while the ratio a = d H /n is held fixed, and define 
R B (a) = lim R B (n,ctn). In the light of the above R(p) ^ R B (&). The 

n-*oo 

Gilbert bound (13b) (Ref. 8, p. 52) tells us that 

Rb($) ^ In 2 + fi In $ + (1 - /§) In (1 - /§) for g $ g |. 
Thus we have 

#(0) ^ln2 + /§ln/§ + (1 - fi) In (1 - j§) - R Ll . (70) 

4.1.2 Bound for Small 13 

Consider a maximum size n-dimensional code with minimum distance 
d, and with M = M (n,d 2 ) code points x x , •••, x M . About each code 
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point x„ construct an open hypersphere in n-space of radius d. Let V„ 
denote the volume of the intersection of this sphere with the n-cube 
[—^-l-j]". Now the union of these M spheres must cover the n-cube: 
for if x„ 6 [—4,+ A]" is not contained in one of the spheres, d (x„, x„) ^ 
d for all v, so that x may be added to the code destroying maximality. 
Thus 

EF^ (2A) n . (71) 

Now let S be the n-dimensional hypersphere of radius d with center at 
the origin, and V n (d) the volume of S [-A,+A] n . It is not hard to 
show that 

V, ;g V n (d), n = 1,2, -,Jf. 
Consequently from (71) 

MV n (d) ^17,^ (2A) n , 

so that 

M(n,d-) ^ [(2A) n /V n (d)l (72) 

Applying the result of Appendix C to (72) yields 

R((3) = lim (1/n) In M(n,4/3n) ^ C (4/3). (73) 

It is shown in Appendix D that for small /3 

R L (0) = Rl, (0) = \ In (A 2 /2ire0) + e(0), (74) 

where «(/3) -* as /3 -* 0. 

fl L (/3) is plotted in Figs. 4(a) and 4(b). 

4.2 Upper Bound on R (@) 

The approach used in this derivation is similar to Plotkin's technique 
for binary codes. We begin by obtaining the following: 

Lemma 4.1: If n < d 2 /2A 2 = d 2 /±A 2 n > |), 

»"»^?4^' (75) 

Proof: Consider the maximum size n-dimensional code with minimum 
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distance d with M = M (n,d ) code points Xi , x 2 , • • • , x M . Let x„ = 
(x»i ,x„2, ■", Xy n ). Then 

(o ) d 2 ^ Z d 2 (x M ,^-tS (»• - ft*) 2 

-g{^s** , -(s«*) f } 

n M 

^ m z Z ft* 2 • 

Since x* ^ A 2 , 

M(M - 1) | = (^) rf2 ^ M ^ 2 . 

from which 

M(n,d 2 )(d 2 - 2A 2 n) ^ d 2 , 
and if n < d 2 /2A 2 , 

"W^- (76) 

completing the proof. 

Lemma 4-2: Let a be an integer not less than two. Then 

M(n,d) ^ aM[n - l,d 2 - {2A/af\. 

Proof: Again consider the maximum size code with M (n,d) points. 
Partition the code into a classes »Si , 6' 2 , • • • , S a , where *S,- consists of 
those code points x„ = (.r„i , x v -> , ■ • -, x yH ) such that 

-A + (i - 1) (2A/a) ^ x ri < -A + i (2A/a), i = 1, 2, •• ., a. 

In other words we partition the interval [ — A,+A] into a subintervals 
of length 2A/a, and assign x„ to class Si according as its first coordinate 
x vl is in the ith subinterval. (To be precise we must close the last sub- 
interval (i = a) at both ends to make the a subintervals cover 
[-A,+A].) 

Now delete the first coordinate from each point in the code. Each 
class Si is now a code of length n — 1 with minimum distance not less 
than 



v^W' 
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Fig. 4(a) (Channel B) — Upper and lower bounds on R(P) vs/3 (0 ^ ^ 0.5). 
Fig. 4(b) (Channel B) — Upper and lower bounds on R(0) vs/3 (0 ^ g 0.2). 

since the first coordinates of two code words in class & do not differ by 
more than 2A/a. 

Further some class Si has at least M (n,d 2 )/a points, so that 

M[n - Xj 1 - {2A/af] ^ (1/a) M(n,<f), 
and the lemma follows. 
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Corollary: Let a < (da/2A ) 2 be an integer. Then 

M(n,d 2 ) ^ a a M[n - a, d 2 - a(2A/a)*\. (77) 

Proof: Inequality (77) follows by repeated application of Lemma 4.2. 
Since by hypothesis d 2 — a(2A/a) 2 > 0, the expression M[n — a, d l — 
a(2A/a) 2 ] is meaningful. 

Derivation of the Bound 

Let n be the greatest integer satisfying 

«. < ^i \f - (» " O (t)1 ' (?8) 

where a ^ 2. Rearranging (78) we obtain 



n < n 



2 d 
a —-7^ ^ 



2A*n 



- 2 



[fe 2 -]- 



-»2?— «=• (79) 



Let us also assume as an additional constraint on a that a > 1//3, 
so that [(2/3a 2 — 2)/ (a 2 — 2)] > 0, and for sufficiently large n,n ^ 1. 
In fact for large n we may approximate n by 



n = n 



Dfe 2 -]- 



(80) 



Now by choice of n (78), < 2A 2 n < [d 2 - (n - n„) (2A/a) 2 ]. Hence 
the Corollary to Lemma 4.2 applies with a = n — n„ yielding 

M(n,d 2 ) ^ a n -"°M[n , d 2 - (n - n )(2A/o 2 )]. (81) 

Also by (78), we may apply Lemma 4.1 to get 



M\n ,d 2 - (n — n ) ( — J ) 



# - (n - n e ) (^Y (82) 



d — (n — n ) ( — j — 2A n 

Thus from (81) and (82) we have 

M(n,d 2 ) ^ a n -"°Q(a,d,n). (83) 

Taking logarithms yields : 
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R(n,d 2 ) ^ In a \l - -1 4- - In Q(a,d,n). (84) 

|_ n J n 

We now let n and d become large while holding the ratio /3 = d 2 /4A 2 n 
fixed. It is easy to show that 

UnQ(a,2AVfa,n) A 0, (85) 

so that using (80) we obtain 

B(fi) ^ In a [l - ^~] = Un «] [^TrJ t 1 " * < 86 ) 

where a is an arbitrary integer satisfying a ^ 2, and a 2 > 1/$. Using 
the choice of a indicated in Appendix E we obtain R (/3) ^ Ru(P) where 

2(ln2)(l -2/8), *£0^i 

(In fc) (1 - 2/3) n 1 — >$zh (87) 



#r,(/3) = 



fcs _ 2 " '* K/ (fc - l) 2 ^ fc 2 

(fc - 3,4, •••)• 

Ru(8) is plotted in Figs. 4(a) and 4(b). For small values of /3, a « 
l/V/3 so we obtain 

fl*(0) = -I In (fc + «(/§), (88) 

where e(/3) — » as /3 -» 0. 

4.3 Bounded Discrepancy Decoding Channel Capacity 

Suppose that for every n, an n-dimensional code is available with 
d 2 /4n = /3. Using BDD we have error probability 

P eB = Pr [rf(x,y) ^ d/2] = Pr [d 2 (xj) ^ fin). (89) 

n 

Since d 2 (x,y) = 23 2 * 2 > where the z k are the (normally distributed) 

4=1 

noise components we have 

Pen = Pr f"g *> ^ jsl. (90) 

By the weak law of large numbers ^2 zf In tends in probability to 
2V(= E(z 2 )). Thus lim P c „ = or 1 according as /8 > N or /8 < N. 

n—<*> 

We define the bounded discrepancy decoding channel capacity de- 
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noted by C B as the supremum of the rates for which it is possible (asymp- 
totically in n) to obtain vanishingly small error probability using BDD. 
From the foregoing we see that C B = R(N). Making use of the bounds 
on R (|Q ) established above we have 



Rl(N) ^C b ^ Ru(N), 



(91) 



where R L and R v are denned by (17) and (18) respectively. These 
bounds on C B are plotted vs the "signal-to-noise ratio" A 2 /N in Fig. 5. 
For large values of the ratio A 2 /N (91) becomes [using (74) and 
(88)] 



1, 1 A 1 . /A 2 \ . n , 1 . A* 



f) 



(92) 



where d , e 2 — » as A 2 /N — > « . 

The channel capacity is the "maximum error free rate" using MDD 
(clearly C ^ C B ). An exact expression for C is not known, however for 
large values of the ratio A 2 /N Shannon 1 has shown that 



_ 1 2 A 2 (A 

C = - In — yp + f3 1 -rr 
2 ire N \N 



(93) 



where e z —* as A 2 /N — » °o . Combining the left inequality of (92) with 
(93 ) we obtain 

C - In 2 + e(A 2 /N) ^C B ^C (94) 




Fig. 5 (Channel B) — Upper and lower bounds on C B vs A 2 /N. 
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where e — > as A 2 /N — > °° . Hence for large signal-to-noise ratio A 2 /N, 
C B differs from C by no more than a constant. Alternately C B /C — * 1 

as .A ViV — > oo . 

4.4 Exponential Behavior of P eB 

For a fixed R < C B , denote by P eB * the smallest attainable value of 
P cB , the error probability using BDD. It was shown above that P eB * -^+ 
0. In this section we shall show that P eB * = exp [— nE B (R) + o(n)] and 
obtain estimates of E B (R). 

Given an n and R, denote by 0„(R) the largest value of /3 attainable 
for a code of length n and with transmission rate R. With R held fixed, 
let /3 (R ) = lim /3„ (/? ) . We can estimate /3 (ft ) in terms of R by 

n-»w 

Rl(P(R)) ^R ^ Ru(P(R)), (95) 

where ft*, and Ru are given (17) and (18) respectively. Inequalities 
(95) result in upper and lower bounds on /3(ft). Thus for any R there 
exists a code (for n sufficiently large ) with minimum distance corre- 
sponding to P(R) (i.e., d = 2 \/fin). With R fixed, this code mini- 
mizes P eB . If code word x is transmitted and y is received, the error 
probability is [from (89)] 

P eB * - Pr[rf 2 (x,y) ^P(R)n]. (96) 

This quantity depends only on the noise and not on x. It is shown in 
Appendix F that 

P rB * = exp [-nE B (R) + o(n)], (97a) 

where 

EAR) = ~2N~ A ~ 2 ln W 6(i{R) = ~2N~ - 2 ~~tf~ ' (97b) 

where /3(ft) = 0(R)/A 2 . The upper and lower bounds on /3(ff) (95) 
yield corresponding bounds on E B (R). These bounds are plotted in Fig. 6 
for the case A 2 /N = 10. 

V. CHANNEL C (GAUSSIAN CHANNEL WITH ENERGY CONSTRAINT) 

5.1 Lower Bound on M (n,8) 

The following bound is similar, though slightly sharper, than the 
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0.8 - 



E B (R) 




Fig. 6 (Channel B) — Upper and lower bounds on the error exponent E B (R) 
vs R (torA'/N = 10). 

lower bound on M(n,9) obtained by Shannon. 11 The derivation used 
here is based on a similar argument in Blachman. 
Let 



SM = 



n-7T 



n/a 



m 



be the surface area of a sphere in Euclidean n-space of radius r, and 
let A„(r,6) be the area of the n-dimensional spherical cap cut from a 
sphere of radius r about the origin by a right circular cone of half 
angle with apex at the origin and axis the semi-infinite line connect- 
ing the origin and the point (r,0,0, • ■ • ). It is not hard to show that 



(n _ 1)y (-i)/« re 



(p dip. 



Derivation of the Bound 

For a given n and 6 consider the maximum size n-dimensional code 
with minimum angle 6 between code points. This code has M(n,6) 
code words. About each code point x, construct the spherical cap cut 
from the surface of the sphere of radius \/nP about the origin by the 
right circular cone with half angle 6 and axis the semi-infinite line 
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joining the origin and x. Thus the cap is the set of points y on the surface 
of the sphere such that the angle a(x,y) < 0. Now the set of all such 
caps (about each of the M code points) must cover the entire surface 
of the sphere. This is so since if x„ is a point on the surface of the sphere, 
and x„ is not on any cap then a (x„ , x) ^ 6 for all code words x, so that 
x may be added to the code destroying the maximality. Since the area 
of each of the M caps is A„ (-\/nP,d), we have 

M-AnWnPfi) ^ S n (VnP) 



or 



/„a\ ^ Sn(VnP) n /- 



I sin n ~ ip dtp V . 



(98) 



This result taken together with Rankin's upper bound 13 yields the 
following estimate of M(n,6): 



<*¥) 



^ V "^) 



■ ["/ sin"" 2 <p d,p\ ^ M (n,6) (99) 

y/*T I — 2 — ) sm ^ tan ^ 



2rf - J / (sin <p) n_2 (cos <p — cos /3) d<p 
where ^ = sin -1 \/2sin (0/2). 

5.2 Asymptotic Estimates of M (n,8) 

For a given n and 0, M (n,0) is the number of points in a maximum 
size n -dimensional code with minimum angle between code points 0. 
Let the corresponding transmission rate be R(n,6) = (1/n) In M(n,0). 
Now with held fixed, let n become large and let 7?(0) = lim R(n,0). 



1092 THE BELL SYSTEM TECHNICAL JOURNAL, JULY-AUG. 1965 

We shall obtain upper and lower bounds on R(d) from the behavior of 
(99) for large n. 
Taking logarithms of both sides of inequality (99 ) yields 



n 



In » Vx + - In ) * ( - I In / sin- 2 p 
n — 1 n ( n + 2 \ n Jo 



- r-,/ „\ ^ li sin i/' tan ^ /- .1 , 
^ ff (n,0) ^ - In *-- • Vtt + - In 



r(!L+i)\ m.h.. 



2 """"V® 

if* 

— - In / sin" -2 <p (cos v? — cos \f/) d\{/, 



where f = sin ' \/2 sin (5/2). 
It is shown in Appendix G that 

lim - In / sin" 2 <p fhp = In sin 5, (101a) 

n-»oo ?l ^(l 

and that 

lim - In / sin" -2 <?(cos ^ — cos ^)rfy> = In sin \f/, (101b) 

from which we obtain (by letting n — > °° ) , 

-In sin 6 £ R(0) ^ - In \/2 sin (0/2). (101) 

The bounds on 22(0) are plotted in Fig. 7. 

5.3 Bounded Discrepancij Decoding Channel Capacity 

We now assume that a code with minimum angle is employed and 
a bounded discrepancy decoder is used. We may assume, without 
loss of generality, that the transmitted word is x = (y/nPfl, • • • ,0). 
The received word y = (VnP +ft,ft, ■■■,!•) will be correctly 
decoded if and only if a(x,y) < 0/2. Since 

, s VnF' (VnP + zi) 

cos a{x,y) = — . — . — , =~o ; ^ 2\1, 

VnPf (VnP + zd + Z 2* ) 
we have 
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R(#) 




40 60 

6 IN DEGREES 



Fig. 7 (Channel C) — Upper and lower bounds on R(0) vs (101). 

\/nP + Zi a/P + (gl/vg) 
cot a = /x ^ >\j — /, \j 

(£*) (iE* f ) 

\n k>i / 
Hence the probability of error is 



[0~| VVP + Zi/>/n 0~ 

cot a ^ cot £ = Pr /n w ^ cot °- 

2J f 1 E *" J 



. (102) 



Now assume that for each h we use a code with minimum an gle 0. 
We shall show that P cB A or 1 according as cot (6/2) < y/P/N 
or cot (6/2) > y/P/N: Recalling that z k (k = 1, • • • ,n) are independ- 
ent normally distributed random variables with mean zero and variance 
AT we obtain 



and 



\/P + Zi/Vn Prob - > VP, 



- £ 2 s Prob - , N. 

n a->i 
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Thus the ratio 

Vp + giAg Prob> A /p (103) 






iV' 



If cot (0/2) < VP/N, then a/P/W - cot (0/2) = e > so that 
from (102) and (103) 



Ps(e) = Pr 



VS 



VP + Zi/Vn 






,o. 



Similarly if cot (0/2) > \/P/N,P B (e) -^ 1. 

Hence we can obtain vanishingly small error probability by choosing 
> 2 arc cot y/P/N or > 2 arc sin (1 + P/N)~\ The bounded dis- 
crepancy decoding channel capacity C B is denned as the supremum of 
rates for which it is possible to achieve P eB -^ 0, or equivalently the 
largest rate for which > 2 arc sin [1 + (P/N)]~ h ; i.e., C B = R{2 arc 
sin [1 + (P/tf )]"*} • Since the channel capacity is C - \ In [1+ (P/#)], 
we may write [1 + (P/A0F* = e_C > hence C « = ^( 2 arc sin e_C )- 
We estimate C B from inequality (101 ) : 

-In sin (2 sin -1 e~ c ) ^ C B ^ -In V2<T C . (104) 

Using sin 2A = 2 sin A cos A, the left member of (104) becomes —In 
2e~ c cos sin -1 e~ c . Since cos sin -1 e~ c = (1 - e~ 2c )\ inequality (104) 
becomes 

C - In 2 - | In (1 - e~ 2c ) fg C B ^ C -\ In 2. (105) 

Inequality (105) is plotted in Figs. 8(a) and 8(b). We see that C B = 
for C ^ J In 2 or P/N ^ 1, and C B /C -> 1 as P/JV -> ». 

5.4 Exponential Behavior of P eB 

In this section we show that for a fixed rate R < C B , the smallest 
attainable probability of error P eB * = exp [-nE B (R) + o(n)], and 
obtain estimates of E B (R). Given an n and P, denote by 0„(P) the 
largest minimum angle attainable for an n-dimensional code with trans- 
mission rate R. With R held fixed, let (R) = lim 0„ (P). From inequality 

n-»« 

(101) 

-In sin 0(P) S R ^ -hi a/2 sin [B(B)/2], 

from which 
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C B /C 
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yS 






0.6 


/ / 






0.4 


1 / 






0.2 


1 / 









l^A 1 i 


i 1 i 


i 



0.346 0.50 I 



Fig. 8(a) (Channel C) — Upper and lower bounds on C B vs C and P/N (solid 
lines). 

Fig. 8(b) (Channel C) — Upper and lower bounds on Cb/C vs C. 



J sin -1 b— ^ d(R)/2 ^ sin -1 if*/y/2). 



(106) 



Thus for every R there exists a code (for n sufficiently large) with 
minimum angle 6(R), where 8(R) is estimated by (106). For such a 
code Pcb is minimized. If code word x is transmitted and y received, 
the error probability is 

fV = Pr[o(x,y) > 0(R)/2]. (107) 

This quantity depends only on the noise (and not on x). Shannon 
[Ref. 11, equation (4)] has obtained an expression for the asymptotic 
behavior of (107), which shows that 

P cB * = exp [-nE a (R) + o(n)} (108) 
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where 



6 = 6(R) 



(109) 



G = 2 



The bounds on 0(R) in (106) yield corresponding bounds on E B (R). 
These bounds are plotted in Fig. 9. 

VI. CHANNEL D (CONTINUOUS CHANNEL WITH AMPLITUDE CONSTRAINT) I 

As in the previous sections we begin by obtaining bounds on M(n,p), 
the maximum number of points in an n-dimensional code with dis- 
crepancy p. 



0.8 v \ 



E R (R) 




^ 



Fig. 9 (Channel C) — Upper and lower bounds on the exponent E B (R) vs R 
for P/N = 4. Upper and lower bounds on E{R) are in dotted lines. 
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6.1 Upper Bound, on R (ft) 

We have defined <S„(x,p) as the region consisting of those vectors 
a € G n for which d (x,a) < p. Applying the Euclidean measure to 
C„ in the obvious way, we set V»(x,p) equal to the volume of S n (x,p). 
Now consider a maximum size 7? -dimensional code with discrepancy p 
consisting of M = M (n,p) points Xi , x 2 , ■ • • , x M ■ Since the regions 
S n (x, , p) about each of the M code points x, are disjoint, 



if 

I 

.=1 



L 7 n (x, , p) ^ volume of C„ = (2A) n . (110) 



Since F„(x,-, p) is independent of x, (due to the homogeneity of e„ 
brought about by wrapping the interval onto the circumference of a 
circle) we set 7 n (x, , p) = V„(p), and (110) yields 

M(n,p) ^ (2A) n /V n ( P ), (111) 

thus 

R(n, P ) = - In M(n, P ) ^ - In ^T ■ < 112 ) 

n n V„(p) 

If we set p = I3n and let n — * °o while /3 is held fixed we obtain 

Rifi) g lim - In J^r = Rvifi). (118) 

» K„(pn) 

It is shown in Appendix C that fln(0) = C (jS) which establishes our 
upper bound. 

6.2 Lower Bound on R (/3 ) 

Again let us consider a maximum size code with discrepancy p and 
M = M (n,p) code words. About each of the code words x, (i = 1, 2, 
• • • , M) consider the region S„(Xj , 2r\p) where 

r(wi 4- w 2 ) ri1A x 

q = sup ' — t— r . (114) 

-i§«l.»l2+J r \ u l) + r ( W 2) 
Jl 

We claim that the union of these regions U <S„(x, , 2rjp) contains e n . 
First let us observe that by definition of 77, 

r(ui + w*) ^ i?[r(wi) + r(w 2 )], 
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so that for x,y,z 6 Cn , 

n _ 

d (x,y) = 2 r(x k - y k ) ^ ^rS r ( x " — 2 *) + £ r(* - y*)l 
i=i L * • J 

= r,d (x,z) + #/ (z,y). (115) 

Now suppose there existed a vector x<, G C„ such that 
u 

x $ U »S n (x,-,27 7 p).Then 
»■— i 

do(Xo,Xi) ^ 2vp (116) 

for each code word x { (i = 1, 2, • • ■ , M). Let a € £„(x,- , p) for some 
code word x, so that d (a,x.) < p, hence from (115) and (116) we have 

d (x ,a)^(l/v)do(x o ,x i ) 

(117) 
- d (x it a) > (l/r,)(2 VP ) - p = p. 

We conclude from (117) that a $ £„(x , p), so that S n (x , p) H 
$ n (x, , p) is empty for all code words x,- , and x,, may be added to the 
code destroying the maximality. Thus we conclude that 

M 

e„ C U S n (Xi,2 VP ). (118) 

1=1 

As in the previous section, let F„(2rjp) be the volume of *S'„(x, , 2r\p). 
From (118) we have 

volume of 6„ = (2A) n ^ M-V n (2r,p), 



or 



MM ^£§0ry (119) 

Again as in the previous section, 

R(fi) * lim iln J 2 ,f" . = R L (0). (120) 

It is shown in Appendix C that Rhifi) = C (2t;/3), establishing our 
lower bound. 

6.3 Bounded Discrepancy Decoding Channel Capacity 

Suppose that for every n, an w-dimensional code is available with 
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discrepancy p = fin (fi fixed). Using bounded discrepancy decoding we 
have error probability 

P cB = Pr [r/,,(x,y) £ p) = Pr K(x,y) ^ /3n], (121) 

where x is the transmitted word and y is the received vector. Since 

n 

d (x,y) = 2Zr(«jt), where the z k are the statistically independent 
noise components, we have 

Pea = Pr yZ rM/n > fi\ (122) 

n 

By the weak law of large numbers, ^3 r(z k )/n tends in probability to 
N(= E(r(z k ))). Thus lim P eB = or 1 according as fi < N or fi > N. 

n-»=o 

We have denned the bounded discrepancy decoding channel capacity 
denoted by C B as the supremum of the rates for which it is possible 
(asymptotically in n) to obtain vanishingly small error probability 
using bounded discrepancy decoding. From the foregoing we see that 
C B = R(N). Making use of the bounds on R (fi) established above we 
have 

C (2 V N) ^ C B ^ C (N) = C, (123) 

where C is the channel capacity (the supremum of those rates for which 
it is possible (asymptotically in n) to obtain vanishing small error 
probability using (optimum) minimum discrepancy decoding). The 
error exponent E B (R) could be estimated exactly as for channel B in 
Section IV. 

Thus it is an open question whether C B is strictly less than the channel 
capacity. In the special case of the quadratic discrepancy where 
r(u) = u , i.e., the case where p(u) = K exp (— Xt* ), it is possible 
to show that C B < C This is done in the following section. 

6.4 The Quadratic Discrepancy 

We now consider the case of the quadratic discrepancy where 
r(u) = w 2 , which corresponds to a noise probability density function 
p(u) = K exp (— \u ), and a discrepancy function 

n 

d„(x,y) = £ (** - Vkf- 
Note that the subtraction x k — y k is performed modulo 2A with the 
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difference reduced into the interval [—A, A], but the squaring and 
summing operations are ordinary arithmetic. 
Let us first observe that 



r(ui -f- ^2) < (ui + u 2 ) _ 1 , 



U\U 2 



r{u x ) + r(u 2 ) " V + M2 2 7 uS + m 2 2 \ " 

Since for any two numbers u* and w 2 2 , the algebraic mean (V + w 2 2 )/2 
is not less than the geometric mean UiU-2 , 

r(ui + u 2 ) gl + 1 = 2 _ 



r(wi) + r(u 2 ) 



Thus, since this value is achieved when V\ = u» £ A/2, rj = 2. The 
lower bound on R(@) (34) is therefore 

R(fi) ^ C (4/3). (124) 

6.4.1 Upper Bound on R(fi) 

Now we establish a new upper bound on R({i) for this special case. 
First we need the following 

Lemma: Let x„ = (x A , x v2 , ■•■ , x m ), v — 1,2, • • • , m, be any m 
points selected from a code with discrepancy p = /3n. Let y be any n-vector 
and let d v = d (x y , y), v = 1, 2, • • • , m. Then 



Y,d v ^ 2(m - l)p. 

Proof: First we show that for 1 ^ p. < v ^ m that 

d,(x,,x„) ^ 4 P . (125) 

To show this consider the vector z 6 6„ : 

( £,1 -f 3„1 X y2 -f 3»2 ^jf^n] 

2 ' 2 ''**' 2 /' 

The addition x vk -j- z,* is, as always, modulo 2A. Clearly 

d (x„, z) = d (x M , z) = I] ( ^^^ )2 = do(X ; , ^ ) . (126) 

Since the regions S„(x„, p) and £„(x M , p) are disjoint, d (x„, z), 
d (x„ , z) ^ p. Thus (126) yields d (x, , x M ) ^ 4p. We now continue 
with the proof of the lemma. 



BOUNDED DISCREPANCY DECODING 1101 

n 

Without loss of generality take y = so that d v = E x„k . Since 

k=\ 

d (x,,x„) ^ 4p 0* < ")- 

^EZ (a** - x *<¥ (127) 

= m X) 2 a »** ~~ 2 (S ^*) 2 = mJldy. 

v k k " >• 

The lemma follows on dividing through by m. We now obtain the 
upper bound on R(0). 

Consider again a maximum size n-dimensional code with discrepancy 
p and with M = M(n,p) code words Xi , Xs , • • • , X* • Consider the 
regions £„(x„ , 2p) about each of the code words x„ (V = 1,2, ••• , M). 
These regions are not necessarily disjoint. At each point y in S n (x* , 2p) 
define a density a (d) : 

a{d) = 2p - d, (128) 

where d is the discrepancy d (x„ , y). The mass of each region is 



= f a(d)dV. (129) 

J d<1p 



If a vector y € C„ belongs simultaneously to the regions about the m 
code points Xi , x 2 , • • • , x m , we assign to y a density equal to the sum 
of the densities contributed by each region; i.e., 

m m 

ay = E <r(d v ) = 2mp- £ d„ , (130) 

where d„ = d (x„ , y). Thus we have 

mass of e„ = / aydV = M(n,p)'n. (131) 

We will bound M(n,p) by finding an upper bound on the mass of e„ . 
By applying the above lemma to (130) we obtain 

<7 y ^ 2mp - 2(m - l)p = 2p. (132) 

Thus 

mass of C„ ^ (2p) (volume of e„) = 2p(2A) n . (133) 
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Applying (133) to (131) yields 

M(n fP ) ^ 2pCU) " . (134) 

Now, 

n = f (2 P - d)dV ^ f dV = V n (2 P - 1) (135) 

J d<2p J d<2p-1 

where V„(2p — 1) is the volume of the region S„(x,2p — 1) (which 
is independent of x). Applying (135) to (134) yields 

2/3w(2A)" 



M(n,pri) ^ 



(136) 



7 n (2/3w - 1) 
where p = fin. Applying the result of Appendix C to (136) yields 

R(fl) = Urn (1/n) In M(n,fin) ^ C„(2fi). (137) 

n-»oo 

This is our upper bound. 

6.4.2 Refinements of Bounds for Large ft/ A 

The upper and lower bounds on R (/3) obtained above are plotted 
vs. P/A 2 in Fig. 10. It can be seen that these bounds diverge for large 




0.05 1/12 0.10 

/3/A 2 



0.15 1/6 0.20 



Fig. 10 (Channel D) — Upper and lower bounds on K(/3) vs 0/ A 2 for quadratic 
discrepancy. 
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values of fi/A 2 . We will now obtain new upper and lower bounds on 
R(0) which in fact converge at /3M 2 = §. 



6.4.2.1 Upper Bound 

A new upper bound on R(fi) will be obtained which will tell us that 
72 08) =0, /3/A 2 > §. First we need the following: 

Lemma: Let a t , a 2 , • • • , a m be a set of real numbers such that 
-A ^ aj ^ +A,j = 1, 2, ■ ■ ■ , m. Then 

Note that, as usual, the difference (a, — a,) is performed modulo 2^4 
with the result reduced into the interval [— A,+A], and the squaring 
and summing operations are ordinary arithmetic. 

Proof: Let us wrap the interval [—A, -\-A] onto the circumference 

of a circle of radius A/t (so that the circumference is 2A). Denote by 

dciai , a,) = \(di — ay) |, 

the circumferential distance between a,- and a, , and by d E (a,- , a,) the 
Euclidean distance between a, and a, (see Fig. 11). It is easy to see that 



, , v A . I d c (a it aj) 

d s (a,- , a y ) = 2 — sin - . 

IT Z {A/IT) 



(138) 




a j = (u jJ Vj) 



Fig. 11 — Diagram illustrating proof of lemma. 
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Since, for ^ x g v/2, sin x ^ (2/ir)x, (138) yields 

d E (ai , ay) ^ (2/tt) d c (a t - , oy). (139) 

Now taking the origin to be the center of the circle, we may assign 
Cartesian coordinates (wy , Vy) to the point a,- , where 

uf + »/ = A 2 //. (140) 

Thus from (139) we have 

2 

E (a,- - ay) 2 = 2 dc 2 (ay, ay) ^ j£ d/fa;, ay). (141) 
i<y \<i 4 »<i 

Since d B 2 (a,- , ay) = («,• - My) 2 + («.• - Wy) 2 , we have 

E (* ■*- *i>* ^iZffe- if/)' + (* - y i) 2 i 

i<y 4 «y 



2 f m 

= x E «(«/ + »/) - GE ^) 2 - GE ^) : 

4 [y-i y y 



■ift-© 



(142) 



• 2 2 

Am 
4 " 

Hence the lemma. 
Derivation of the Bound 



Suppose we have a maximum size code with discrepancy p and 
M = M(n,p) code words x„ = (x vl , x v2 , ••• , x yn ), v = 1, 2, • • • , M. 
We have shown [inequality (125)] that d(x^ , x„) ^ 4p (p. ^ v). Thus, 
making use of the above lemma, we have 

( o ) 4 P = E d o( x *> x m) = E E («»* ■*■ s,*) 2 



i 2 n .^2 i2,/2 



- tt 4 4 



so that for = p/n > Ay 8, 

M = M(n, P ) £ 



8 P - A 2 n ' A 2 ' (143) 
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Hence, 

R{n, P ) = - In M(n, P ) S - In — L^. 

Letting n — > °° with /3 held fixed we obtain for 0/A > f , 
72 (0) = lim#(n,/3w) = 0. 



(144) 



(145) 



In a manner similar to that used in Section IV we can use (143) 
to obtain the following bound on R (/3) valid for /3/A 2 < \: 

R(S) < 9 (In 3) [1 - (8/3/A 2 )]. (146) 

As is evident from Fig. 12, inequality (146) does not yield much im- 
provement in our upper bound, hence the derivation is omitted. 

6.4.2.2 Lower Bound 

A new lower bound on R(P) will now be obtained. This bound is 
always sharper than the previously obtained bound R(0) ^ C„(4:N), 
however the best improvement is for large /3/A 2 . 

Suppose that we require that x k be one of the following m points on 
the interval [— A,-\-A], where m is an even integer: 



RW 




0.05 



O.IO 1/8 0.15 

0/A 2 



Fig. 12 (Channel D) — Refined upper and lower bounds on R(0) vs P/A 2 for 
quadratic discrepancy. 
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Such a code certainly satisfies the requirements set forth in Section II. 
In the exactly the same manner that the previously derived lower 
bound R(J3) ^ C (4/3) was obtained, we can show that 

R(fi) ^ c m m) 

where 

C m (£) = In AK m (£) - fr m (it), (148) 

where X m (£) is defined by 

E ***** = s E •"*•■. ( 149a ) 

k * 

and K m (%) is 

K m (& =[Ze-^ )Uk V, (149b) 

where the Uk are the m points of (147). 
Since no value of m yields a uniformly strongest bound we write 

R(0) ^ max C m (40). (150) 

mevon 

This new bound is plotted in Fig. 12. Let us observe that the lower bound 
R(0) ^ C 2 (40), and the upper bound R(P) ^ 9 In 3[1 - (8/3/A 2 )] 
agree when (3/ A 2 = |. Thus C fl = for /3/A 2 ^ |. 

6.4.3 Estimation of C B 

We now obtain an estimate of C B for the case of a quadratic dis- 
crepancy function. As discussed above C B = R(N). The bounds on 
ff (0) of (137), (146), (150) yield 

[C o (2A0, 

maxC m (4JV) ^ C fl g min< / sat\ (151) 

9 In 3 



(-so- 



Since the channel capacity C = C (N) > C„(2N), the first upper bound 
of (151) implies that C B is strictly less than the channel capacity C. 
For large values of the "signal-to-noise" ratio A 2 /N, the left side of 
(151) may be approximated by C o (4A0- We can make use of the asymp- 
totic form of C (£) obtained in Appendix D: 

C.(f) =|ln 0A7«ft) +«(*>> (I 52 ) 
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where *(£) — > as £ — » 0. Applying (152) to (151), we obtain 



* 1 »£n + "(£) g C ° g * '"^T + »Cf). (153) 

where «i , e 2 — * as A 2 /N — > « . Further since the channel capacity C 
is (for large A 2 /N) 

C = C.W - i In l£ + *(£), (154) 

where e 3 — > as A /N — > °o , (153) may be rewritten as 

C - ln2 + e 5 (A 2 /AO ^C B ^C-| In 2 + «,(A 2 /tf), (155) 

where e B , e 6 — » o as .<4 2 /JV — * °° • Thus for large values of A 2 /A7 (and hence 
C), the bounded discrepancy channel capacity C B differs by no more 
than a constant (In 2) from the channel capacity C. Thus the ratio 
C B /C -» 1 as A 2 /N -> oo . 

Let us remark at this point that the channel capacity of the Gaussian 
channel with amplitude constraint has been shown by Shannon 1 to be 
approximately C (N) (for large A 2 /N), which is the same as the 
capacity of the present channel. This fact lends plausibility to the 
claim that the present channel is an approximation to the Gaussian 
amplitude constrained channel for large values of A /N . 

APPENDIX A 

In this appendix we show that for any function r(u) for which 
r(w)— >0asw— >0, and for any £ satisfying 

< £ ^ ~ I r(u)du, (156) 

there exists a unique X(£) which satisfies 

f r(u)e- X(i)rM du = £ f e- M&rM du. (157) 

•'O •'0 

For channel B we are interested in the case r(u) = u 2 , however for 
channel D we need this proposition for arbitrary r(u). If we define the 
function £(X) by 

J r(u)e-* rM du 
$ft) = -^72 . ^ X < co, (158) 

/ -Xr(u) , 

/ e clu 
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it will suffice to show that 

(a) £(X) is strictly monotone decreasing, 

(b) {(0) = j [ r(u)du, 

(c) lim £(X) = 0. 

X-»oo 

If (a), (b) and (c) are true, £ (X) is a one-to-one mapping of the half 
line [0, «. ) onto the interval 

(°'if r(! H- 

(a) To show that £(X) is monotone decreasing, consider 

«m _ -tf''S(/>S + (/, v S ; 



d\ 



(159) 



by the Schwarz inequality, 

(f re'^duj < (f* rV* r «fo) (f e~" r du^ , (160) 

(the strict inequality holding). Thus d£(X)/dX < and (a) is established, 
(b) f(0) =/ r(u)du/ J du = JJ r ^ du - 

(c) (due to H. O. Pollakf) since £(X) is monotone decreasing and 
positive for X < *>. we know that lim g(X) = |8 ^ 0. If = (c) is 

F X-M 

established. Thus we assume the contrary, i.e., /3 > 0. Since £(X) is 
monotone decreasing we have £(X) ^ 0, all X < <». Thus for any A, 

f £(X)dX ^ 0A. (161) 

•'o 

Now let us observe that £ (X ) may be written 

£(X) = -^(ln/ o ^" Xr(u) ^). (162) 

t An alternate proof was given to the author by L. A. Shepp. 
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Substituting (162) into (161) we obtain 
•a M 



f £(A)d\ = -In f e~ Ar(u) du + In A ^ /3A. (163) 

Jo •'n 



'0 •'O 

Or, 



i f e~ hrM du ^ e~ p \ (164) 

A Jo 

Dividing through by e~ p/i we have 

1 f A e+ AO-r(«„ du g L (165) 

A Jo 

Now since r (w) — > as u — ► 0, choose 5 sufficiently small so that r (u) < 
/3/2 whenever ^ u ^ 8. Equation (165) now becomes 

u|f e +K( '- r) du 
A Jo 

> i f e +X{p - Hu)) du (166) 

A Jo 

^ 1 A0/2 / j 5 A/3/2 

~ A Jo A 

Now (166) holds for all A < oo. Thus we need only choose A large 
enough so that 

4 e Km > 1 

to deduce a contradiction. Thus (c) follows. 

APPENDIX B 

Proof That rj is Finite 
Define the function 

, , r(ui + u 2 ) ,. c -s 

g{ui,u 2 ) = —. — r— I r-^r , (167) 

r(ui) + r(w 2 ) 

where —A ^ ih , th ^ ~\~A and (i*i , wO ^ (0,0), and the function 
r(u) is given by (27). Note that the addition Wi -j- w 2 is performed 
modulo 2A. We must show that 77 = sup g(ui , w 2 ) is finite, or that 
g(ui , w 2 ) is bounded. 
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By assumption (8d), r(u) is continuous, and by assumptions (8c) 
and (8d) r(u) > when u ^ 0. Thus g(ui , Us) is continuous over its 
domain. If g is unbounded, let Ci/i ( " ) ,W" ) )"=i be a sequence such that 
0(m 1 (b) ,u 2 ( " ) )» -» °°. Then it is easy to see that (wi (B) ,Wa (n) ) -» (0,0). 
Thus to show that -q is finite we need only show that g is bounded in the 
neighborhood of the origin. 

Now let Rx = ( (ui , u 2 ) : Uy , u 2 ^ 0} . We shall show that 

g(ui,u 2 ) = sup 0(1*1,1*2). (168) 



77 = sup 

— *S "l."2^+'* 



(U,, «,)€«! 



If (wi , Ma) $ i?i , then either 1*1 and i* 2 are both negative or u x and u 2 
have opposite signs. In the first case g{u\ , u 2 ) = 0( — 1*1 , — u 2 ), where 
( — i*! , —1*2) 6 #1 • In the second case say | t*i | ^ | t*s |, then by as- 
sumption (8d) and (8c) r{u x -j- ih) ^ r(ui). Thus 0(1*1 , 1*2) ^ r(u x )/ 
[r(ui) + r(u 2 )] ^ 1 = g(A,0) where (A,0) € R\ . Thus we need show 
only that g is bounded in the neighborhood of the origin where 
Ml , u 2 ^ 0. 

With i*i and u 2 sufficiently small, the addition 1*1 -j- u 2 = 1*1 + i* 2 . 
Also by assumption (8e), we may write 



r(u) = au (1 + «(i*)), 

where a > 0, a > 0, and e (1*) — ► as 1* — * 0. Thus 

0(1*1,1*2) 

a (1*1 + t* 2 ) a (1 + c(mi + 1*2)) 
" a(i*i)«(l + e(tti)) + aw 8 «(l + e(i* 2 )) 



(169) 



(170) 



(1*1 + u 2 ) a 
l*i° + Mi a 



1 + e(l*i + U>) 



1 + 



Ml 



1*1° + Ul° 



(u0 + 



u, 



1*1° + 1* 2 Q 



:(l* 2 ) 



Now, 



so that 



Mi 



Ma 



Ui a + U 2 a ' Mi a + u 2 ° 



gl, 



where €1(1*1 ,1*2) — > as t*i , i* 2 — > 0. Thus, since 



(171) 



(172) 
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(1 + - 2 Y 

ill = \ w < o a-1 



q < (Ui + U 2 ) a _ \ Ml/ < ga-l (173) 

Wi a + U 2 a 1 + (U2/Ui) a " 

we conclude that is bounded in the neighborhood of the origin, and 
therefore that 17 is finite. 

Let us remark at this point that discrepancies r(u) do exist for which 
rj = co . For example, r(u) = exp ( — \/u ). If we set Mi = u 2 and let 
Mi — > we obtain 

-l/{2«!)« 

(«!,«*) = — ^"T^^- (174) 

In this case, of course, r (u) does not satisfy (169) so that p (u) does not 
satisfy (8e). 

APPENDIX C 

For channel B, let V n (p) be the volume of the intersection of a sphere 
in Euclidean n-space of radius p and center at the origin with the cube 
[— ^4.,A] n . For channel D, V n (p) is the volume of S n (x,p) = volume of 
S n (0,p), where 

S„(0,p) = < a = («, , a 2 , m , • • • , a n ) 6 C„ : d o (0,a) = 2 r(a k ) < p>. 

In this appendix we evaluate 

1 (0 4)" 

lim - In -^- = E a . (175) 

We shall find E a by solving an equivalent probability problem: Let 
Xi , X 2 , ■ ■ ■ be a sequence of random variables uniformly distributed 

on the interval [ — A.+A]. Let Y„ = T) r (X k ). For channel B, r (u) = u*. 

i=l 

It is clear that 

Pr IY, < Pi = gf„ . <176) 

hence 

-lim (1/n) In Pr [Y n < an] = E a . (177) 

n-»oo 

We now make use of 
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Chernoff's Theorem*: Let Z x , Z 2 , ■■■ be a sequence of independent 
identically distributed random variables with moment generating 

function #(exp (Z,0) = M(t). Let P„ = Pr TJ Z, ^ an , where 

a g E(Z,). Then 

1 n . 

- In P„ * In m, 

n 

where m = min e~ a 'M(0- 

If we let Zi = r(Xi) where X, is the above random variable, then 

Y n = E Z < • Thus from ( 17? ) Ea = _ln m - 
t-i 

The moment generating function of Z, is 

M{t) =E[e z < 1 ] = ± f_ + \' wt dx, 

so that 

fit) = e-'M (0 = i •-' f " e rW '(te, (178) 

and 

m = min /(/) . 

To minimize /(0, let us differentiate (178) with respect to t: 

d M = = — e~ at \ C A r(x)e rM 'd.r - a f " e rU)t dx] , 
dt 2A \_J-a J-A J 

so that 

f +A r(x)e r(x)l dx = a ]** e r(x "dx. (179) 

The solution of (179) for t is t = - X(a) where X(£) is denned by 
(19a). With t so chosen 

1 r+ A 

Wj\ l +aX(o) / — r(x)X(a) , 

so that 

In m = - In 2AK (a) + aX(a) (180) 
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where K (a) is defined by (19b). Thus 

E a = -In m = In 2AK(a) - a\(a) = C (a), (181) 

where C (a) is defined by (19). 

If we apply this result to (73) (channel B) with a = 4/3, we obtain 
R L2 ((3) = C (4/3). If we apply this result to (113) and (120) (channel 
D) with a = /3 and 2tj/3 respectively, we obtain Ru(P) = C o (0) and 
R L (P) = C (2t7/3) respectively. Finally, applying this result to (136) 
(channel D with quadratic discrepancy) with a = 2/3 yields R(@) ^ 
0,(2/3). 

APPENDIX D 

Estimate of C(£) for Small £ with r(u) = u 2 

We first obtain an estimate of X (£) for small £ and then show how this 
estimate can be used to estimate C(£). 
The quantity X(£) is defined by (19a): 

f A-^Ai = £ f e - 2X( *W (182) 

•'O J 

Observe that X(£) monotonically approaches infinity as £ — > 0. Chang- 
ing the variable of integration in (182) we obtain 

r\^\A ,„ rV2\A „ 

/ sV ,2 dx = 2X£ / e-" 2 dx. (183) 

Integrating the left integral by parts yields: 

., ViXA fV2\A __ fy/2\A 

-xe- tln + / e- xi,2 dx = 2H / e~ x2,2 dx. (184) 

•'0 •'0 

Rearranging terms we obtain 

X - i (1 - m(X)) (185) 

where 



m(X) - 



VaUr"' 



r 



r^ 



(186) 



/2x^ 

'II 

Since /* (X ) ^ we have an upper bound on X : 

X ^ 1/2$. . (187) 
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To obtain a lower bound on X set 

A = (1/2$) - X. (188) 

From (185) 

M (X) X M (X) __ Ay/2\*e* A% 



A = 



"» l -"^ f^-iV»- (189) 



D(X)- 

It may be verified by differentiation that for X ^ 3/ (2A 2 ) the numerator 
iV (X ) is monotonically decreasing and the denomination D (X ) is mono- 
tonically increasing so that A is monotonically decreasing. With X = 
3/(2A 2 ) we obtain by substitution into (189) A = 0.76/A 2 and by 
substitution into (185), £ = 0.22A 2 . Thus for $ ^ 0.22A 2 

Returning to (189), we may write 

\2* A* J 



A < 



Z)(3/2) 



A(1.12)e" x2/2t n (191) 

0.76 



= 1.35A 



-i-A,i[i-?^q. cm 



Thus we have for J ^ 0.22A : 

2£ " = 2$ 
Since the quantity C (£) is defined by 

C.(?) = ln2AK.(f) - *X(*), (193) 

where 



K.(& = 



f^e-^duj 1 , (194) 



we could then use the upper and lower bounds on X(£) of (187) and 
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(192) to obtain an estimate of C„(£). However, this turns out to be a 
very cumbersome procedure and we shall side-step this chore. Suffice 
to observe that X(£) approaches l/(2£) very rapidly as £ approaches 
zero so that for small £ we could take X to be 1/ (2£) and obtain 



9 A* 

C.(Q =* In ^r + €(*), 

7re£ 



(195) 



where e (£) — » as £ — » 0. 



APPENDIX E 

Completion of Derivation of Upper Bound on R{0) for Channel B 
Inequality (86) expresses the fact that 

R<$) £/(a)(l-2/§) (196) 

where 



/(«) = 



- 2 



In a 



(197) 



and a is any integer satisfying a ^ 2, a > 1//3 (0 ^ $ < £). To 
obtain the tightest bound we seek to minimize f(a) subject to these 
constraints. It may be verified by differentiation that f(a) is a mono- 
tone increasing function for integer values of a for a S: 2. Thus to 
minimize /(a) we choose a as the smallest integer satisfying a ^ 2, 
a > 1//3. Thus we choose 



and 



a = 2 when § ^ fi ^ i, 



a = A: when 



(ft - D s 



1 

fc 2 ' 



(fc= 3,4,5, •••)• 



APPENDIX F 



Estimate of E B (R) for Channel B 

Equation (96) expresses the fact that 

P* = Pr [g tf 

where the 0,- are independent normally distributed random variables 



> an 



(198) 
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with mean zero and variance N, and a = /3(R) = A $(R). We seek an 
expression for E B {R) = lim — (1/ra) In P eB *. We again make use of 

n-»oo 

a form of: 

Chernoff's Theorem 16 : Let F x , F 2 , • • • , F„ be independent and identi- 
cally distributed random variables with moment generating function 

E[exp (F.01 = M{t). Let P n = Pr |g F, ^ tm\, where a ^ # (F,). 

Then 

lim (l/n)P„ = In m, 

n-»oo 

where 

m = mm. e" 01 M (t) . 



i>0 



If we set F, = *" then E s = lim - (1/n) In P„ = -In m. The 

n-»co 

moment generating function is 

It may be verified by differentiation that the quantity e~ at M(t) is 
minimized at t = (1/22V) — (l/2a) (which is positive if a > N). Thus 



m = 



-[-«&-£)>(*-£)■ 



Setting a = A 2 p(R) and taking logarithms we obtain 

_£(&) 1. aB(fl) 
2tf 2 AT ' 



fl99) 



APPENDIX G 



Completion of Asymptotic Estimates for Channel C 

1. Let I n = I sin" -2 ^ d<p. We must show that 
Jo 

E = lim - In /„ = In sin 5. This is (101a) . 

n-oo U 
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r e _ 

(a) /„ = / sin" 2 d dtp = (9) sin" 2 0, so that 

1 1 n — 2 n 

- In /„ ^ - In 6 + In sin > In sin 0. 

n n n 

(b) h ^ f e sin"" 2 <p ^ ^ sin" -2 (d - -J \-\ , so that 

- In I n = In sin ( — - ) + - In - — » In sin 0. This completes 

w n \ nj n n 

the proof. 



2. Let I n = I sin" 2 ^ (cos ^ — cos yf/)d<p. We must show that 



E = lim - /„ = In sin rff. This is (101b). 
n-»oo n 



(a) I n ^ I sin" -2 \^(cos <p — cos $)(kp — sin" -2 ^[sin \f/ — \(/ cos \p], 
•Jo 

so that - In /„ ^ In sin ^ + - In [sin ^ — \f/ cos rp] > In sin \l/ 

n n n 

. sin" -2 ^>(cos <p — cos \f)d<p 
^ sin" -2 ( f ) / ., (cos <p — cos \f/)d<p 

a /** 

Now 1=1. (cos <p — cos ^) rf<p 



(200) 



-(*-*)- 



= sin tf/ — sin ( \p — ~) cos \p 



v • & $ 

= sin \1/ — sin \l cos — r cos \l/ sin - — - cos \1/. 
n n n 

Expanding sin (^/n) and cos (\(//?i) into power series in (\f//n) we ob- 
tain 



/ = sin 



«»*[£ + .(£)]-£*.*a + .<U>. 
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Thus 



i2 



- In I = - In t- sin * + - In (1 + o(l)) -^> 0. 
n n 2n 2 n 

Thus from (200) we have 

I In 7 n ^ Vl^I l n S in U - t\ + I In / -^U In sin *. 
n w \ n/ n 

Thus £ = ln sin ^ which completes the proof. 

APPENDIX H 

!T/ie Capacity of Channel D 

The channel capacity is defined 1 by 

C = max[H(y) - H(y\x)], (201) 

2d) 

where x is the input digit, y the output digit and H(y\x) the conditional 
uncertainty of y given x. The maximization is performed over the input 
distribution p(x). Since y = x + z,H{y\x) = H(z) so that 

p(u) \np(u)du, 

independent of p(x). Now H (y) is maximized when the random varia- 
ble y is uniformly distributed on [—A,+A]. Due to the symmetry of 
the channel, this occurs when p{x) = 1/(2A), -A ^ x ^ +A. In 
this case 

Thus the channel capacity is 

C = ln 2 A + J p(w) ln p(w)du. (202) 



Writing p{u) = 2f exp [— Xr(w)] we obtain 

(203) 



C = ln 2A + In if J_ p(w)dw - X £ r(u)p{u)du 



= ln 2AK - XiV, 
where N is defined by (32) or 
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+A 



N = [ r(u)K e^ r(u) du. (204) 

J- A 

Also, since p(u) integrates to unity, 

1 = j_ +A K e-* r{u) du, (205) 

we have 

£ r(.u)e***du - N [ «~ XrW du l (206) 



with N and r(u) specified, X may be found as the solution to (206). 
With X so specified we may find K a from (205), thus 

C = \n2AK (N) - N\(N), 

where X (A/) is the solution of (206) and K„(N) is the solution to (205). 
This is the same as C = C„ (N) where C (£) is defined by (19). 

GLOSSARY OF SYMBOLS 

The following symbols are used throughout the paper: 

n = dimension of input, output and noise vectors. 

x = (xi , X2 , • • • , x n ) = input vector or code word. 

y = (//i , 7/2 , • • • , ?/n) = output vector or received vector. 

z = (zi , z-> , • • • , z n ) = noise vector. 

M = number of words in a code. 

R = (1/n) In M = transmission rate. 

p ei = probability that the receiver makes an incorrect decoding 

decision when code word i is transmitted (i = 1,2, • • • , M). 
it 

P e = (1/M) ^Z Per = over-all error probability. 

»=i 

MDD = minimum discrepancy decoding (always optimum for the 

channels considered in this paper). 
BDD = bounded discrepancy decoding. 
P C M = error probability (P„) using MDD. 
Pcb = error probability using BDD. 
C = channel capacity = "maximum error free rate" using MDD. 
E(R) = the best attainable error exponent using MDD, (at rate R). 
C B = bounded distance decoding channel capacity, or 
"maximum error free rate" using BDD. 
E B (R) = the best attainable error exponent using BDD (at rate R). 
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The following symbols are used in connection with specific channels : 

Channel A 

q = the number of symbols in the input, output and noise alpha- 
bets. 
p = the probability that the channel transmits a given symbol 
correctly. 
d»(u,v) = the Hamming distance between two n-vectors u and v = the 
number of positions in which u and v differ. 
C(p ) = \nq- H(p„) — p„hi(q— 1) = channel capacity of channel 

A with symbol error probability p a . 
H(p) = -p In p - (1 - p) In (1 - p) = the entropy function. 
d = the minimum distance between code words. 
e = (d — 1 )/2 = number of correctable errors in a code with 
minimum Hamming distance d. 
M (n,d) = maximum number of code words in an re-dimensional code 

with minimum Hamming distance d. 
R(n,d) = (l/n)M(n,d) = rate corresponding to M(n,d). 
/3 = d/2n, a ratio appearin g in our bounds. 

t = [( q - l)/q0\ [1 - Vl - [2g/(g - 1)]0], another quantity 
appearing in our bounds, 
[a;] = largest integer not exceeding x. 
R(0) = limit R(n,20n), asymptotic form of R(n,d). 

n-»oo 

<*{p,Po) = p In (p/po) + (1 - p) In [(1 - p)/(l - p.)], a quantity 
appearing in our error bounds, 
s = parameter defined by R = In q — H(«) - s In (q - 1) = 
C(s) 

Channel B 

A = maximum amplitude of input coordinates. 

N = variance of normally distributed noise coordinates. 

d E (u,v) = ]>^ (uc — Vi) 2 = Euclidean distance between the 
i=i 

n-vectors u and v. 

d = the minimum distance between code words. 

M(n,d 2 ) = maximum number of code words in an n-dimen- 

sional code with minimum Euclidean distance d. 

R(n,d i ) = (l/n)\nM(n,d 2 ) = rate corresponding to M(n,d). 

= d 2 /4:n $ = 0/A 2 , ratios appearing in our bounds. 
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R((l) = lim R(n,4(in), asymptotic form of R (n,d 2 ). 

n-»oo 

Rl(P) and Ru(P) = lower and upper bounds on R(0) given by (18) 
and (19) respectively. 

The function C„($), < £ < A 2 /S, is denned as follows: X ($) is the 
quantity denned by 

[ A r(u)e- X(i)r(u) du = it [ A e- Mi)r{u) du 
Jo J o 



where r(u) = u , and 



Then 



C.(f) -ln2AX.tt) -*X(*). 



Channel C 



P = (1/n) X the energy of a code word. 
AT = variance of the normally distributed noise coordinates. 
d K (u,v) = the Euclidean distance between u and v. 
a(u,v) = the angle between w-vectors u and v. 

6 = the minimum angle between code words. 
M (n,6) = maximum number of code words in an n-dimensional code 

with minimum angle 0. 
R(n,6) = (l/n) In M (?i,d) = rate corresponding to M(n,8). 

ip = sin -1 \/2 sin (6/2), a quantity appearing in our bounds. 

Channel D 

6„ = set of real n-vectors u = (u\ , Uo , ■ ■ • , u n ) satisfying | m* | ^ 

A. 
-]-,— = addition and subtraction modulo 2A (with result reduced 

into the interval [— A,+A]). 
p(u) = noise probability density function. 
r(u) = (1/X) In \p(0)/p(u)] (-A g u ^ +A), quantity related 

to the discrepancy. 

n 

d (u,v) = ^r(u k — v k ) = discrepancy between n-vectors u and v 

belonging to e„ . 
AT = E(r(z)), a parameter associated with the noise density p(u). 
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Co (£) , defined exactly as for channel B but with the appropriate 
r (u) used instead of u . 

■n = sup —. — * . . , a quantity appearing in our bounds. 
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